Non-destructive processing of digital image data

ABSTRACT

Systems for rendering Effect graphs for non-destructively processing digital image data which integrate Central Processing Unit (CPU) processing and Graphics Processing Unit (GPU) processing are provided. Additionally provided are systems for processing digital image data utilizing Effect graphs. The systems of the present invention integrate CPU processing and GPU processing to facilitate accelerated rendering of Effect graphs and, consequently, accelerated processing of digital images. Methods for processing digital image data utilizing the systems herein described are also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

The subject matter of this application is related to the subject matterof U.S. patent application Ser. No. ______ filed Jun. 24, 2005, entitled“Performing a Pre-Rendering Pass in Digital Image Processing” (attorneydocket number MFCP.120060), U.S. patent application Ser. No. ______filed Jun. 24, 2005, entitled “Caching Digital Image Data” (attorneydocket number MFCP.120061), U.S. patent application Ser. No. ______filed Jun. 24, 2005, entitled “Accumulating Transforms Through an EffectGraph in Digital Image Processing” (attorney docket number MFCP.120062),and U.S. patent application Ser. No. ______ filed Jun. 24, 2005,entitled “Translating Layers into Effect Graphs in Digital ImageProcessing” (attorney docket number MFCP.120063), each of which isassigned or under obligation of assignment to the same entity as thisapplication, and incorporated in this application by reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable.

BACKGROUND

Video Cards or Graphics Processing Units (GPUs) differ from CentralProcessing Units (CPUs) in that they utilize very high speed memory andcouple it with a specialized processor that is capable of handlingsimple calculations on pixel data very efficiently. Video cards are notgeneral purpose processors. However, they can compute, for instance,which surfaces should be visible to a user and which should be hidden aswell as what textures should be mapped onto those surfaces to make themlook realistic in a three-dimensional (3D) scene much more rapidly thana CPU could accomplish the same task. This is largely due to the factthat while CPUs can adequately perform a variety of general tasks, theyare not optimized for any particular operation whereas video cardsutilize very specialized hardware to perform only a single function at atime.

Video Cards (GPUs) have traditionally been used for two primarypurposes. First, GPUs have been utilized to accelerate simple graphicsoperations on a user's screen. For instance, when a user scrolls througha document displayed on a display device (e.g., a conventional computermonitor), a large number of pixels are moved around. Due to thecomputational demand such action places, traditional CPUs are notexceptionally rapid or efficient at accomplishing this task. As such, aGPU may be utilized in conjunction with the CPU to achieve more rapidand efficient scrolling.

Secondly, GPUs have been utilized to render simple primitives (e.g., tofill an area with a color, draw a line, or the like). As withacceleration of simple graphics operations, GPUs can more rapidly andefficiently render such simple primitives than traditional CPUs.

More recently, video games have started to become more popular causingthe demand for increased realism and speed in such games to increase. Assuch, video game manufacturers have begun to tackle complex issues ofvideo rendering, for example, rendering a three-dimensional scene whileproviding shading, reflections, texture mapping, and the like to makethe scene look as realistic as possible while still rapidly respondingto user input.

Outside of the particular areas mentioned above, video cards have notbeen utilized to accelerate CPU functions.

Use of digital images in arenas outside of video game manufacture hasalso enjoyed a recent increase. However, as digital images get bigger(more megapixels) and pixels get deeper (at least 16 bits per channel),they are straining the computational resources available in traditionalCPUs resulting in slower, less efficient digital image processing.

SUMMARY

The present invention relates to non-destructive editing of digitalimage data utilizing, at least in part, Graphics Processing Unit (GPU)memory. Non-destructive editing (or non-destructive processing) refersto editing (or processing) wherein only that image data for whichrendering is requested will be rendered at the time of the request. Inother words, rendering takes place beginning from unalteredoriginally-loaded image data on-demand whenever an image is rendered.Non-destructive editing utilizing GPU memory permits the loading, intothe GPU, of only the bits needed for rendering on-demand. As such,significantly higher performance rendering is permitted relative totraditional editing models that utilize only Central Processing Unit(CPU) memory.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The present invention is described in detail below with reference to theattached drawing figures, wherein:

FIG. 1 is a block diagram of an exemplary computing environment suitablefor use in implementing the present invention;

FIG. 2 is a schematic diagram of an exemplary blending Effect havingthree inputs and two outputs, in accordance with an embodiment of thepresent invention;

FIG. 3 is an schematic diagram of an exemplary Effect graph thatselectively blurs areas of an input image as directed by a mask, inaccordance with an embodiment of the present invention;

FIG. 4 is a schematic diagram of an exemplary Effect graph, inaccordance with an embodiment of the present invention;

FIG. 5 is a schematic diagram of an exemplary Effect graph with a countshown for each Effect, in accordance with an embodiment of the presentinvention;

FIG. 6 is a schematic diagram of an exemplary Effect graph illustratinga mask associated with a blur Effect, in accordance with an embodimentof the present invention;

FIG. 7 is a schematic diagram of an exemplary Layer for use in an Effectgraph, in accordance with an embodiment of the present invention;

FIG. 8 is a schematic diagram of an exemplary fully-connected Layer ofan Effect graph, in accordance with an embodiment of the presentinvention;

FIG. 9 is a flow diagram illustrating a method for processing digitalimage data utilizing an Effect graph, in accordance with an embodimentof the present invention;

FIG. 10 is a flow diagram illustrating a method for processing digitalimage data utilizing a vertically-oriented Effect graph in accordancewith an embodiment of the present invention;

FIG. 11 is a flow diagram illustrating a method for processing digitalimage data utilizing a vertically-oriented Effect graph in accordancewith an embodiment of the present invention;

FIG. 12 is a flow diagram illustrating a method for processing digitalimage data utilizing an Effect graph in accordance with an embodiment ofthe present invention;

FIG. 13 is a flow diagram illustrating a method for processing digitalimage data utilizing an Effect graph in accordance with an embodiment ofthe present invention;

FIG. 14 is a flow diagram illustrating method for creating an Effectgraph for processing of digital image data in accordance with anembodiment of the present invention;

FIG. 15 is a flow diagram illustrating a method for processing digitalimage data utilizing an Effect graph in accordance with an embodiment ofthe present invention.

FIG. 16 is a flow diagram illustrating a method for processing digitalimage data utilizing an Effect graph, the method of FIG. 16 representingadditional processing steps that may take place after the method of FIG.15, in accordance with an embodiment of the present invention

FIG. 17 is a flow diagram illustrating a method for processing digitalimage data utilizing a vertically-oriented Effect graph in accordancewith an embodiment of the present invention; and

FIG. 18 is a flow diagram illustrating a method for processing digitalimage data utilizing a vertically-oriented Effect graph in accordancewith an embodiment of the present invention.

DETAILED DESCRIPTION

The subject matter of the present invention is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies. Moreover,although the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

Embodiments of the present invention provide systems and methods forprocessing digital image data utilizing Effect graphs. Having brieflydescribed an overview of the present invention, an exemplary operatingenvironment for the present invention is described below.

Referring to the drawings in general and initially to FIG. 1 inparticular, wherein like reference numerals identify like components inthe various figures, an exemplary operating environment for implementingthe present invention is shown and designated generally as computingsystem environment 100. The computing system environment 100 is only oneexample of a suitable computing environment and is not intended tosuggest any limitation as to the scope of use or functionality of theinvention. Neither should the computing environment 100 be interpretedas having any dependency or requirement relating to any one orcombination of components illustrated in the exemplary operatingenvironment 100.

The invention is operational with numerous other general purpose orspecial purpose computing system environments or configurations.Examples of well known computing systems, environments, and/orconfigurations that may be suitable for use with the invention include,but are not limited to, personal computers, server computers, hand-heldor laptop devices, multiprocessor systems, microprocessor-based systems,set top boxes, programmable consumer electronics, network PCs,minicomputers, mainframe computers, distributed computing environmentsthat include any of the above systems or devices, and the like.

The invention may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, etc., that performparticular tasks or implement particular abstract data types. Theinvention may also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices.

With reference to FIG. 1, an exemplary system for implementing thepresent invention includes a general purpose computing device in theform of a computer 110. Components of computer 110 may include, but arenot limited to, a processing unit 120, a system memory 130, and a systembus 121 that couples various system components including the systemmemory to the processing unit 120. The system bus 121 may be any ofseveral types of bus structures including a memory bus or memorycontroller, a peripheral bus, and a local bus using any of a variety ofbus architectures. By way of example, and not limitation, sucharchitectures include Industry Standard Architecture (ISA) bus, MicroChannel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnect (PCI) bus also known as Mezzanine bus.

Computer 110 typically includes a variety of computer-readable media.Computer-readable media can be any available media that can be accessedby computer 110 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes both volatileand nonvolatile, removable and non-removable media implemented in anymethod or technology for storage of information such ascomputer-readable instructions, data structures, program modules orother data. Computer storage media includes, but is not limited to, RAM,ROM, EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by computer 110. Communication media typicallyembodies computer-readable instructions, data structures, programmodules or other data in a modulated data signal such as a carrier waveor other transport mechanism and includes any information deliverymedia. The term “modulated data signal” means a signal that has one ormore of its characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared and other wireless media. Combinations of any of the aboveshould also be included within the scope of computer-readable media.

The system memory 130 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 131and random access memory (RAM) 132. A basic input/output system (BIOS)133, containing the basic routines that help to transfer informationbetween elements within computer 110, such as during start-up, istypically stored in ROM 131. RAM 132 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 120. By way of example, and notlimitation, FIG. 1 illustrates operating system 134, applicationprograms 135, other program modules 136, and program data 137.

The computer 110 may also include other removable/non-removable,volatile/nonvolatile computer storage media. By way of example only,FIG. 1 illustrates a hard disk drive 141 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 151that reads from or writes to a removable, nonvolatile magnetic disk 152,and an optical disk drive 155 that reads from or writes to a removable,nonvolatile optical disk 156 such as a CD ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,digital versatile disks (DVDs), digital video tape, solid state RAM,solid state ROM, and the like. The hard disk drive 141 is typicallyconnected to the system bus 121 through a non-removable memory interfacesuch as interface 140, and magnetic disk drive 151 and optical diskdrive 155 are typically connected to the system bus 121 by a removablememory interface, such as interface 150.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 1, provide storage of computer-readableinstructions, data structures, program modules and other data for thecomputer 110. In FIG. 1, for example, hard disk drive 141 is illustratedas storing operating system 144, application programs 145, other programmodules 146, and program data 147. Note that these components can eitherbe the same as or different from operating system 134, applicationprograms 135, other program modules 136, and program data 137. Operatingsystem 144, application programs 145, other programs 146 and programdata 147 are given different numbers here to illustrate that, at aminimum, they are different copies. A user may enter commands andinformation into the computer 110 through input devices such as akeyboard 162 and pointing device 161, commonly referred to as a mouse,trackball or touch pad. Other input devices (not shown) may include amicrophone, joystick, game pad, satellite dish, scanner, or the like.These and other input devices are often connected to the processing unit120 through a user input interface 160 that is coupled to the systembus, but may be connected by other interface and bus structures, such asa parallel port, game port or a universal serial bus (USB). A monitor191 or other type of display device is also connected to the system bus121 via an interface, such as a video interface 190. In addition to themonitor 191, computers may also include other peripheral output devicessuch as speakers 197 and printer 196, which may be connected through anoutput peripheral interface 195.

The computer 110 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer180. The remote computer 180 may be a personal computer, a server, arouter, a network PC, a peer device or other common network node, andtypically includes many or all of the elements described above relativeto the computer 110, although only a memory storage device 181 has beenillustrated in FIG. 1. The logical connections depicted in FIG. 1include a local area network (LAN) 171 and a wide area network (WAN)173, but may also include other networks. Such networking environmentsare commonplace in offices, enterprise-wide computer networks, intranetsand the Internet.

When used in a LAN networking environment, the computer 110 is connectedto the LAN 171 through a network interface or adapter 170. When used ina WAN networking environment, the computer 110 typically includes amodem 172 or other means for establishing communications over the WAN173, such as the Internet. The modem 172, which may be internal orexternal, may be connected to the system bus 121 via the networkinterface 170, or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 110, orportions thereof, may be stored in a remote memory storage device. Byway of example, and not limitation, FIG. 1 illustrates remoteapplication programs 185 as residing on memory device 181. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

Although many other internal components of the computer 110 are notshown, those of ordinary skill in the art will appreciate that suchcomponents and the interconnection are well known. Accordingly,additional details concerning the internal construction of the computer110 need not be disclosed in connection with the present invention.

When the computer 110 is turned on or reset, the BIOS 133, which isstored in the ROM 131, instructs the processing unit 120 to load theoperating system, or necessary portion thereof, from the hard disk drive141 into the RAM 132. Once the copied portion of the operating system,designated as operating system 144, is loaded in RAM 132, the processingunit 120 executes the operating system code and causes the visualelements associated with the user interface of the operating system 134to be displayed on the monitor 191. Typically, when an applicationprogram 145 is opened by a user, the program code and relevant data areread from the hard disk drive 141 and the necessary portions are copiedinto RAM 132, the copied portion represented herein by reference numeral135.

As previously mentioned, embodiments of the present invention providesystems and methods for utilizing the acceleration capabilities of avideo card or GPU in digital image processing. Such methods and systemsmay utilize, e.g., video interface 190 of FIG. 1. It will be understoodand appreciated by those of ordinary skill in the art that a “digitalimage”, as the term is utilized herein, refers to any digital image dataincluding a static and/or dynamic digital image and that any and allcombinations or variations thereof are contemplated to be within thescope of the present invention. Utilizing the systems and methods hereindescribed, pixel data may be transformed in a variety of ways at anaccelerated pace, that is, faster than the CPU could do it itself.

Embodiments of the present invention additionally integratenon-destructive editing with GPU acceleration. “Non-destructive editing”or “non-destructive processing”, as those terms are utilized herein,refer to editing (or processing) wherein only that image data for whichrendering is requested will be rendered at the time of the request, thatis, without flattening or saving the data to intermediate steps. Inother words, rendering takes place beginning from unalteredoriginally-loaded image data on-demand whenever an image is rendered.

Prior applications have attempted non-destructive editing in combinationwith CPU processing to varying degrees. However, it has not been fullyrealized as CPU processing is sufficiently slow that the performancecould not match that of the flattened models on the relatively smallimages that were prevalent at the time. More recently, however, verylarge (megapixel) photo images have increased in popularity making anon-destructive editing model more appealing.

Utilizing a GPU in combination with non-destructive editing, ascontemplated by embodiments of the present invention, makes imageprocessing hugely-often orders of magnitude-higher performance thantraditional CPU-based flattened models. Indeed, flattened models areill-suited for GPU implementation given the relatively small memoryfootprint of the GPU (flattening is very memory intensive) and limitedbandwidth for reading image data from the GPU to CPU memory.

Non-destructive editing in combination with a GPU permits the loading,into the GPU, of only the bits needed for rendering on-demand. Hence,the need to ensure that all of the original pixels are processed at fullresolution for each edit is avoided.

Methods and systems in accordance with embodiments of the presentinvention integrate GPU acceleration capabilities into a CPU digitalimage processing model. That is, the GPU is integrated into the digitalimage processing model architecture, as opposed to being providedperipherally as merely a special feature or the like. In this way, CPUand GPU processing are unified so that users (e.g., programmers ordevelopers) who are working in digital image processing caninterchangeably use the two modalities for what their strengths orcapabilities are with the target device, for example, without having tochoose either CPU processing or GPU processing alone.

It should be noted that while integrated into the system architecture,use of a GPU is provided as an optional component. Thus, if therendering device includes a capable video card, the system may call onits acceleration capabilities as needed or desired. However, if thesystem does not include a capable video card, it may still be utilizedfor digital image processing.

Methods and systems of the present invention provide the functionalityfor creating an Effect graph and wiring it using a GPU, a CPU, or somecombination of the two. The digital image processing model architecture,in accordance with embodiments of the present invention, is comprised ofelements referred to herein as Effects. An Effect, as that term isutilized herein, is a basic image processing class. That is, Effects arebasically pixel operators. They take in buffers and pixel data,manipulate the data, and output modified pixels. For instance, asharpening Effect takes in image pixels, sharpens the pixel edges, andoutputs an image that is sharper than the image pixels taken in. Inanother example, an exposure Effect takes in image pixel data, adjuststhe color balance of the pixels, and outputs and image having modifiedcolor balance from what was taken in. The primary function of an Effectis thus to process pixels.

Different Effects, e.g., masking, blending, rotating, and the like, maybe defined to implement a variety of image processing algorithms. Asmore fully described below, users are permitted to wire pre-definedEffects together to achieve a desired result. If desired, a user mayalso define new Effects to implement interesting image processingalgorithms. Effects can process any number of input images and produceany number of output images. The number of inputs and outputs isdetermined at construction time and cannot be changed without recreatingthe Effect. The meaning of each input or output is defined by the Effectitself. In most cases, the meaning is order-dependent.

By way of example only, FIG. 2 illustrates a schematic diagram of ablend Effect having three inputs and two outputs. This particular blendEffect blends its first two inputs (Input 0 and Input 1), modulated by amask supplied to its third input (Input 2). In this example, the firstoutput (Output 0) produces a color image, while the second output(Output 1) produces an image in shades of gray. Many Effects produceonly one output.

In a way, Effects act like integrated circuits in that a user mustunderstand what each Effect does before they may be wired together. Likeintegrated circuits, a collection of Effects wired together can achievecomplex results that exceed the sum of the parts.

When Effects are wired together, they form an Effect graph. An Effectgraph, as the term is utilized herein, is a Directed Acyclic Graph(DAG). It's “directed” because all connections imply a direction of dataflow (from the output of one Effect to the input of the next). It's“acyclic” because no loops are allowed, that is, no path through thegraph causes an Effect's output to come back as one of its inputs. It'sa “graph” rather than a “tree” because the output of an Effect can feedthe inputs of multiple Effects. Effect graphs connect multiple Effectstogether to achieve a complex output built up from multiple simplerbuilding blocks.

With reference to FIG. 3, a schematic diagram of an exemplary Effectgraph 300 is illustrated that selectively blurs areas of an input imageas directed by a mask. Notice how the image Effect object 310 suppliesimage data to both the blur Effect object 312 and the blend Effectobject 314. This would not be possible in a tree structure.Additionally, there are no cycles in the Effect graph 300. That is, theoutput of, for instance, the blend Effect object 314 is not the input ofone of the other Effect objects (image Effect object 310, blur Effectobject 314, or mask Effect object 316). The model of the presentinvention automatically checks for cycles when an Effect graph isrendered and/or changed to prevent infinite loops.

When an Effect is called upon to render, some interesting things happenwith each Effect. First, each Effect determines if the request can besatisfied using an existing cached buffer. While caching strategies arediscussed more fully below, for now it is sufficient to contemplate thatno work needs to be done if the output of that work already exists in anappropriate cache. If the request can be partially satisfied by anexisting cache buffer (this happens frequently, for instance, when auser is panning through an image or re-sizing the window), overlappingpixels are copied from the cache to a new image buffer and then theremaining pixels are rendered and copied into the appropriate parts ofthe new buffer.

Effects may be rendered using two different methods: a required rendermethod which operates the CPU (software render) and an optional rendermethod which operates the GPU (hardware renderEach Effect determineswhether it should attempt to render in hardware (GPU) or software (CPU).The choice can be mandated by the user (there is a software-render-onlyoverride associated with each Effect), by the application, and/or by thecurrent context (a software-render-only override in the context allowsthe client or any Effect higher in the graph to force software renderingof the sub-graph). If no software override is mandated, the base classwill attempt to render the Effect in hardware and, if possible, an imagebuffer containing the requested result will be returned. If the Effectcannot be rendered in hardware, a software rendering will take place.

Note that there are also dynamic conditions that might influence whichrendering method is called. For example, an Effect might call for ahardware render but catch an out-of-memory exception an attempt is madeto allocate temporary buffers that exceed the available memory of thevideo card. The default response is for the Effect to call for asoftware render (CPU) to get the results it needs. That is, the modelcan fall back to the CPU if it runs out of GPU memory. In this way,performance can be optimized based on resource availability.

An Image buffer represents image (pixel) data stored in system memory(CPU memory), video memory (GPU memory), or both and is passed from oneEffect to another through the Effect graph. The actual location of thedata is generally not important to the user as the user may simply askfor the data desired for processing at the location s/he desires toprocess it and the Image buffer will automatically copy the data to thatlocation, if necessary. That is, Image buffers hide the details of wherethe pixel data is stored and automatically make it available in theappropriate location for the Effect.

An image buffer is the main class used by Effects to read and writeimage data. Image buffers generally have a maximum size that is quitesmall in comparison to the size of images a user may desire to render.For instance, in one embodiment, the maximum width or height dictated byhardware limitations may not exceed 2K pixels in either dimension. Aparticular Effect graph might not even be able to render 2K pixels in asingle pass. The maximum size depends on the particular Effectscontained in the graph. For example, a large-radius blur Effect mustread many more pixels than it writes, so the maximum buffer size it canproduce is reduced by the amount it over-samples its input.

Users cannot easily anticipate these limitations when making a renderingrequest. As such, a user simply asks for what they need. However, it isa possibility that the returned buffer will be of a different size thanwas requested.

An instance of an image buffer holds memory that is allocated and cannotbe reclaimed. Image buffers are useful to hold image data that is beingactively read or written by an Effect.

Sometimes, an Effect might like to cache image data between renderingpasses. However, this data cannot be reclaimed if held in an imagebuffer. If video memory runs low as the rest of the Effect graph isrendered, the engine must revert to software rendering unless it canreclaim sufficient resources.

The Image cache object is an alternate way to cache image data. Theimage cache object allows an Effect to keep a reference to image datathat it would like to reuse during subsequent rendering, but it couldregenerate if it had to. By holding an image cache object and releasingthe image buffer object, the Effect allows the memory to be reclaimedwhen video memory gets tight. The goal is to allow other Effects to usehardware (GPU) acceleration rather than reverting to software (CPU)rendering in low memory scenarios. A user attains an image cacheinstance from any image buffer s/he may want to read at a later time.Subsequently, the image buffer may be released while the image cachecontinues to be held. This maintains a reference to the image data andallows reclaiming of the contents thereof if necessary.

The organization of Effects into graphs, as opposed to trees, presentssome challenges during rendering. For example, contemplate an Effectgraph such as the one illustrated in the schematic diagram of FIG. 4. Inthe illustrated Effect graph 400, there are four ways for image data toflow from the bottom image Effect object 410 to the output of the topblend Effect object, blend Effect B 418. First, image data may flow fromimage Effect object 410, to blend Effect object A 414, to blend Effectobject B 418. Second, image data may flow from image Effect object 410,to levels Effect object 412, to blend Effect object A 414, to blendeffect object B 418. Third, image data may flow from image Effect object410, to levels Effect object 412, to blend Effect object A 414, to blurEffect object 416, to blend Effect object B 418. Fourth, image data mayflow from image Effect object 410, to blend Effect object A 414, to blurEffect object 416, to blend Effect object B 418.

While four passes through the Effect graph may not seem to beunmanageable, the combinatorial possibilities are exponential. If a fewmore Effects are coupled with additional blend Effects, the number ofunique paths through the graph grows to hundreds or thousands. A naiverendering model must render Effects near the bottom of the graph manytimes, producing the same output each time. This exponential growth andrepeated rendering of identical Effects leads to highly inefficient andslow digital image processing.

The model of embodiments of the present invention alleviates some, ifnot all, of this excessive rendering by including automatic buffercaching at each Effect. For instance, if the blend Effect object A 414in the Effect graph of FIG. 4 holds a cache, it may simply return thecache each subsequent time it is asked to render, rather than making twomore requests to level Effect 412 and image Effect 410 to render itsinputs. In this example, the image Effect object 410 is only asked forits output twice rather than four times.

More particularly, in order to get from raw image to final output usingan Effect graph, each Effect along the path must be queried for itsoutput. However, an Effect can't render its output until its inputbuffers are rendered. As such, upon being asked for its output, eachEffect queries its inputs (i.e., queries for the image buffers of itsinput Effects so that it can operate on them). However, an Effect'sinputs can't render their output until their respective input buffersare rendered and, accordingly, must query its own inputs. This queryingprocess continues all the way from the top of the Effect graph (wherethe final output is rendered) to the bottom where the input (e.g., rawimage, JPEG, etc.) is rendered. Additionally, this process must becompleted for each query by following one potential path through theEffect graph at a time. For instance, in the above-described exampleutilizing the Effect graph of FIG. 4, while blend Effect object B 418only has two inputs, each of those inputs must query its inputs alongtwo different paths to render its output. Each Effect produces an imagebuffer for each potential path and provides the final Effect (in thisexample, blend Effect object B 418) with one of the needed inputs foroutputting the final output.

However, when caching is utilized by the intermediate Effects in theEffect graph (e.g., levels Effect object 412, blend Effect object A 414,and blur Effect object 416), each intermediate Effect may hold or cachethe query results computed during the most recent prior time the Effectgraph was rendered. In this way, when a query is initiated at the top ofan Effect graph, once an intermediate Effect that is common between thetwo paths being examined is reached, it is not necessary to query allthe way back to the bottom of the Effect graph. That is, the query onlyneeds to progress back to the first Effect that is common between thepaths being examined. Locating this common Effect between two paths ofan Effect graph and caching or holding information at that point,alleviates the potential exponential calculations discussed above.

In order to determine at which Effects a cache should be held during aninitial pass through the Effects graph (and to prevent unnecessarycaching at Effects that will not be further queried), a pre-renderingpass of the graph is performed before rendering is begun. Thepre-rendering pass identifies, before any actual computations areperformed on the pixels, what area of the Effect graph is going to bequeried and identifies those portions of that area that are going to beasked for multiple times during the rendering, as well as the number ofquerying instances that may be expected. Then, upon rendering, anyEffect identified as requiring multiple queries may cache the queriedinformation in an associated image buffer until it has been called onthe maximum number of times. At that point, the image buffer may bereleased as the information contained therein is not going to be askedfor again.

In the example illustrated in FIG. 4, blend Effect object 414 is queriedfor image buffers with two different sizes as the blur Effect object 416reads more pixels than it writes. In this case, it is fortunate that thelarger image buffer is requested first, and the second request may besatisfied without further rendering. However, if the larger image bufferrequest came after the smaller image buffer request, the request wouldnot be as easily fulfilled.

To avoid inadequate performance when rendering requests come inincreasing order of size, the model of the present invention providesfor the performance of a pre-rendering pass. The pre-rendering pass wasdiscussed hereinabove with respect to identifying portions of the Effectgraph which will be subject to multiple queries for the sameinformation. During the pre-rendering pass, each Effect also accumulatesa bounding rectangle that contains similar rendering requests. Thus,during subsequent rendering, an Effect may produce an output buffer thatis larger than necessary for the current request, anticipating use ofthe same cache to satisfy further requests at no additional cost.

These image caches avoid one problem, but cause another in that a lot ofvideo memory is begin reserved to hold cache buffers, some of which maynever be used again. The simple solution would be to reclaim cachebuffers on a least-recently-used basis whenever an attempt to allocate acache buffers is made causing an out-of-memory exception to be returned.But that means that video memory is frequently being run at fullcapacity (which is undesirable for any other user of the video memory),and it incurs the overhead of exception processing on nearly everyallocation.

With reference to FIG. 5, a schematic diagram of an Effect graph with acount shown for each Effect is illustrated and designated generally asreference numeral 500. Each Effect has a counter associated with each ofits outputs that counts how many connectors are connected to thatoutput. Each time an Effect is finished using one of its input buffers,it calls on the corresponding input Effect to release the output bufferand the counter for that output is decremented. When it reaches zero,the Effect assumes that no further request for that output buffer willbe received during the current rendering pass, so the Effect releasesits output cache image buffers are held only during a single renderingpass as it is undesirable to hold any more information than necessarydue to limited video card (GPU) memory.

Even when an Effect has a cache, it must still tell its input Effects torelease their output buffer caches, even though it doesn't need them torender. This is necessary to ensure that Effect counters are properlyadjusted (and consequently, caches are properly released).

After rendering is complete, the top Effect will hold the only cache inthe graph. Actually, as will be discussed more fully below, there areother reasons a user might want to keep additional caches to accelerateinteractive rendering. The present example is provided herein simply toillustrate how caching may be used to prevent exponential rendering.

To accelerate interactive rendering, it is useful to keep caches forparts of the Effect graph that haven't changed from one rendering passto the next. The way a user typically builds up a project is one Layeror Effect at a time (Layers are discussed more fully below). The userwill, for instance, add a Effect to adjust the exposure of an image andthen add another Effect to adjust the temperature of the image and thenadd a third Effect to sharpen the image. Due to this step by step addingof Effects, the user generally adjusts only the most recently addedEffect at a time, e.g., the user will complete all exposure adjustmentsbefore adding the temperature Effect. However, more that one exposureadjustment may be necessary, each adjustment requiring a rendering pass.

To accommodate this typical project scenario and avoid unnecessaryre-rendering of those portions of the Effect graph that haven't changedsince the previous rendering pass, the Effect base class includes a“dirty flag” that is set when an Effect has changed since the lastrendering pass. During a pre-rendering graph traversal (pre-renderingpass), dirtiness is propagated upwards through the Effect graph,disposing caches on any path between a dirty Effect and the root of thegraph.

Dirtiness is used to keep caches for parts of the graph that aren'tchanging. It is assumed that any dirty Effect is likely to be dirty forthe next rendering pass as well. This is the case when the user isadjusting a parameter value or otherwise modifying one Effect at a time.

For example, suppose that a user is painting on the mask associated witha blur Effect in the Effect graph illustrated in the schematic diagramof FIG. 6. During the pre-rendering pass, the top blend Effect, blendEffect B 618, detects that it has two input Effects that are not dirty(blend Effect A 614 and blur Effect 616) and a third input Effect whichis dirty (mask Effect 620). Whenever an Effect detects this dirtinessdiscrepancy between its inputs, it assumes that the mask Effect 620 isthe only input that is going to be modified on the next rendering passand instructs the non-dirty input Effects to hold onto their caches. Inthe illustrated scenario, blend Effect B 618 would instruct blur Effect616 and blend Effect A 614 to hold onto their caches and would instructmask Effect 620 to release its cache. (Note that image Effect 610 andlevels Effect 612 will not be queried on the next rendering pass as theyprovide the inputs for blend Effect A 614.) This selective caching maybe referred to as “cross-render caching” and the performance improvementin terms of accelerated interactive rendering is significant as only theblend Effect B 618 and mask Effect 620 need to do real work for eachrendering pass.

As previously mentioned, the benefits of cross-render caching areespecially interesting given that most users build their projects oneLayer or Effect at a time, and they usually adjust the last Layer orEffect they added, which is usually at the top of the Effect graph. Inthe dynamic rendering model of embodiments of the present invention, theuser is free to change any Layer or Effect at any time. Withcross-render caching, adjustments near the top of the Effect graph willbe faster than those near the bottom.

Note that with cross-render caching, caches are held across renderingoperations for particular digital image data (i.e., a single image)rather than only during a single rendering pass as was the case withrespect to caching for the purpose of avoiding exponential growth ofprocessing operations.

The concept of “Layers” in Effect graphs has been alluded to a number oftimes herein. In the Effect graph model of embodiments of the presentinvention, Effects are the “atomic” image processing elements. Layerscombine Effects in useful ways that address interesting scenarios.Layers construct Effect graphs, but are not directly involved inprocessing or rendering them.

To understand why Layers are helpful, consider an adjustment layermodel. Contemplate that various different types of adjustments aresupported, but all of them support a mask and a blending mode. This isaccomplished by separating the masking and blending functionality intoindividual Effects, and then wiring several simple Effects together toachieve the complex result desired to be achieved. However, when thereare repeated editing units, users have to wire the same Effects togetherover and over, and it's difficult to develop localized intelligence tooptimize the graph. For example, the mask Effect can be omitted if themask is empty, and the blend Effect can be omitted if there is noblending mode and no mask

One might think of Layers as smart macros that simplify the creation ofstandard image editing pipelines. In fact, the concept of a pipeline isintroduced by Layers. A Layer takes a single image input and produces asingle output. Of course, the Effects within a Layer can be organized inarbitrarily complex sub-graphs.

A Layer takes any Effect and adds support for arbitrary blending modescontrolled by an optional mask. In one example, this looks like theschematic illustration shown in FIG. 7, the Layer being designatedgenerally as reference numeral 700. Each Layer owns its Effects andwires them together dynamically (i.e., at run time) to create its partof the Effect graph. It can apply smart dynamic optimizations to thegraph. For example, since a Layer receives events when its Effectschange, if it notices that the blend Effect 710 mixing parameter haschanged to its minimum value, it might exclude the blend Effect and themask Effect 712 from the Effect graph it builds.

The Effect graph of FIG. 7 shows the most common way the blend Effect710, mask Effect 712, and main Effect 714 may be wired together.However, the Layer supports more complex relationships among these threeEffects. In some cases, the mask Effect may be automatically generatingmask data based on the Layer's input image. The main Effect may or maynot depend on input image data. In some cases, the main Effect may needaccess to mask data.

A fully-connected Layer may look like the schematic diagram of FIG. 8.Of course, Layers can get even more complicated than what is shown inFIG. 8. Layers can replace any of the Effects shown in the diagram withmultiple Effects wired arbitrarily.

An Effect that outputs a single-channel image buffer can serve as a maskEffect. Masks may be constant pixel-based masks (i.e., single-channelimage Effect) or constant vector-based masks (i.e., that rasterizesvector-based data into a single-channel output buffer. Additionally, insome situations, a mask may be generated dynamically from input data.Generally, this would be accomplished by a user's dynamic selection ofpixels that satisfy some criteria, e.g., a particular color range.

For instance, with reference to the mask Effect of FIG. 8, suppose auser desires to lighten the shadows in a particular image coming up thepipeline. Accordingly, s/he adds an exposure Layer into the Effect graphand places a mask on that Layer such that the mask will be placed onlyin the shadowed regions of the image. Generally, masks are painted on orpasted in based on pixel selections and, accordingly, they are static.However, in accordance with embodiments of the present invention, thepixel data and the masks can be made dynamic by including an instructionthat allows the user to, e.g., select the sky for darkening. This wouldbe accomplished by the user's dynamic selection of pixels that satisfythe criteria of the particular color range of the sky. Subsequently,whenever the pipeline is changed (e.g., by the addition of a rotationLayer), the mask is automatically re-calculated rather than having to bere-painted on by the user.

In cases where automatic corrections such as dynamic masking can be donein the GPU, this is a very compelling scenario because the mask will begenerated at the same resolution as the input image. Unlike constantmasks, no rasterization of vectors or interpolation of image samples isrequired.

One of the main advantages of dynamic masking is that dynamic maskspermit a user to make changes to the pipeline and have the masks adaptto those changes where appropriate. An example would be red-eye. Red-eyemay be corrected by de-saturating the region of the pupil using a maskto locate the pupil on a de-saturation Effect Layer. Setting the maskstatically (algorithmically using a selection on red pixels or by hand)would fail if the underlying image was moved (translated, rotated orscaled, for instance). A dynamic mask, however, can re-detect where thered pupils were in the image and re-set the mask.

The Layer class expects blend Effects to have three inputs (two imagesto blend plus a mask). Each blend algorithm is represented by a separateEffect subclass. There will generally be enough common code to warrant acommon base class. When the user changes a Layer's blend mode, theprevious blend Effect is discarded, and the new blend Effect takes itsplace as the Effect graph is rebuilt.

No blend Effect is required (the Layer's output is simply the output ofthe main Effect) if no mask Effect is connected to the blend Effect; afully-opaque mask is assumed.

A “single-input, single output” Layer model significantly simplifies theprocess of building Effect graphs. An additional benefit of Layersbecomes apparent when a user wishes to save a configuration of apipeline they've created. With an Effect graph, saving is actually arather difficult task to accomplish. Each Effect would have to be askedto serialize itself (i.e., save its settings and the like) and, usingthat information, the Effect graph would have to be scribed into, e.g.,XML. The Layer model makes this simpler as the rendering engine itselfis aware that the graph contains a number of Layers and can describeeach of them. The Layers are then asked to describe themselves. As eachLayer has its respective graph built internally, that information doesnot need to be stored.

An additional advantage of Layer serialization over Effect serializationis that Layers are in trees and Effects are in graphs which can be (andgenerally are) DAGs. A greedy depth-first recursive serialization of atree may be performed (e.g., at each node, an Effect can query itschildren to describe themselves and then describe itself). In a DAG,this runs exponentially, for all the same reasons discussed hereinabovewith respect to caching to prevent rendering from going exponential inEffect DAGs. As such, it is much cleaner to address this issue at theLayer level than at the Effect level.

In the dynamic digital processing image model in accordance withembodiments of the present invention, where and when data is transformedis fairly important, both in terms of quality and performance. Tounderstand the quality concerns, suppose a user wishes to rotate animage using a conventional image processing model. Once it is determinedthat the image is in the desired orientation, the rendering engine mustre-sample the image at the new rotated value and produce a new set ofpixels from that data. If the user then rotates the image again, theconventional rendering engine does not remember where the image startedbefore the prior rotation. As such, it simply rotates the image again(unaware that it has been rotated before) and, upon completion,re-samples the image and produces a new image again. This process can goon an on as additional rotation corrections are performed, with the databeing re-sampled and a new image created each time. The challenge withthis approach is that every time it is re-sampled, the data degrades.

In the dynamic digital processing image model in accordance withembodiments of the present invention, however, all adjustments aredynamic. Thus, when a user rotates an image, the rendering engineremembers the cumulative amount it has been rotated. That is, each timeit is asked to make a rotation modification, it goes back to theoriginal data and applies the rotation. Therefore, when an image isregenerated subsequent to a rotation modification, it is re-sampled onlyonce from the original image data. This significantly reduces the amountof degradation the image must endure.

Further, this behavior may be preserved, even when there are differenttransformations at different levels of the project that are each addingsome rotation. The accumulated product of all transforms throughout theEffect graph are summed and applied just once. This application isusually performed near the bottom Layer of the pipeline as there is amechanism in the rendering Engine that accumulates all transformationsand pushes them as far down the pipeline as possible before they areapplied. In this manner, transformations are not performed any moreoften than necessary and on the smallest number of pixels needed toproduce an accurate render.

It should be noted that there are a couple of things that can stop atransformation from being pushed through the Effect graph. For instance,an Effect that doesn't understand how to operate on pixels that havebeen transformed may be encountered and, thus, that particular Effecthas to operate on the un-transformed image, the transformation beingapplied only after the Effect has been applied.

To accommodate for situations such as this, during the pre-renderingpass, each Effect informs the engine what kinds of transformations itcan handle and what kinds of transformations it cannot handle, as morefully described below.

As previously stated, when transformations take place is also importantfrom a performance perspective. In the Effect graph pipeline, it is mostefficient to process as little data as possible. Suppose a user hasloaded a very large image (e.g., one billion pixels) and they desire tobegin editing that image. Instead of operating at the resolution of theimage (e.g., instead of operating on one billion pixels), the dynamicrendering model of embodiments of the present invention operates at theresolution the user is actually viewing, e.g., the two million pixels orso at which the display device being utilized outputs the image. It isdesirable to knock down the resolution of the image at the earliestpossible moment in the pipeline so that every Effect that is dealingwith the image thereafter is dealing with a resolution of the data thatis considerably smaller. As such, the Effects can operate on the imagemuch faster and use significantly less GPU or CPU image memory. Itshould be noted that a scaling factor is being applied to the data topermit this result. However, just as previously discussed with regard toquality concerns, there may be Effects which cannot handle a scaling,that is, Effects that do not know how to deal with scaled data.Accordingly, during the pre-rendering pass, each Effect will inform theengine if it cannot handle this type of transformation and the scalingwill not take place until that Effect has already processed thenon-scaled data.

It should be further noted that the effect of pushing transformations asfar down the pipeline as possible may result in a particulartransformation not taking place at the location that a user hasspecified if the rendering engine determines that it can actually beapplied at another transformation location down lower in the Effectgraph.

By way of example only, imagine that a user has produced an Effect graphwith a blur Effect (blur with a pixel radius of five) and then adown-sample or scaling Effect (by a factor of ten). The user wouldexpect the blur radius to be applied and then the image scaled. Inreality, however, as the down-sample Effect is a transformation, if itcan be pushed down the graph, it will be applied to the sample first andthen the blur will be applied to the lower resolution image. Note thatthe blur Effect will have to be modified in this instance to, e.g., ½pixel blur for the overall effect to be the same from the user's(viewing) perspective. If the blur Effect declared to the renderingengine that it could not handle blurring a scaled image, the scalingEffect would not be pushed below it in the pipeline.

Turning now to FIG. 9, a flow diagram is illustrated which shows amethod 900 for processing digital image data utilizing an Effect graphin accordance with an embodiment of the present invention. Initially, asindicated at block 910, a plurality of Effect objects are received.Effect objects have been discussed extensively hereinabove and mayinclude, by way of example only, a blend Effect object, a mask Effectobject, a blur Effect object, and the like. Once the plurality of Effectobjects have been received, an Effect graph is wired into an Effectgraph utilizing at least a portion of the plurality of Effect objects,as indicated at block 912. The Effect graph may be rendered utilizingany of CPU memory, GPU memory and a combination of CPU memory and GPUmemory as necessary and/or desired.

Subsequently, as indicated at block 914, digital image data to beprocessed is received. The digital image data is then processedutilizing the Effect graph, as indicated at block 916. In oneembodiment, processing the digital image data may includenon-destructively processing at least a portion of the digital imagedata received. Next, the processed digital image data may be stored inat least one of the CPU memory and the GPU memory in association with atleast one image buffer. This is indicated at block 918.

It should be noted that image buffers in accordance with embodiments ofthe present invention are capable of being passed between the GPU andthe CPU automatically and without user intervention. Additionally, imagebuffers in accordance with embodiments of the present invention arecapable of being converted between a first pixel data format and asecond pixel data format automatically and without user intervention.

With reference to FIG. 10, a flow diagram is illustrated which shows amethod 1000 for processing digital image data utilizing avertically-oriented Effect graph, the digital image data being processedfrom a bottom of the Effect graph to a top of the Effect graph, inaccordance with an embodiment of the present invention. Initially, asindicated at block 1010, a vertically-oriented Effect graph having a anEffect object located at a junction initiating two divergent paths isreceived. Subsequently or simultaneously, digital image data to beprocessed utilizing the Effect graph is received, as indicated at block1012. Next, as indicated at block 1014, a first processing pass of theEffect graph from the bottom to the top thereof is performed along afirst of the two divergent paths.

During processing, at least a potion of the digital image data is cachedin an image buffer associated with an output of the Effect object. Thisis indicated at block 1016. Subsequently, a second processing pass ofthe Effect graph is performed, the second pass extending from the imagebuffer associated with the Effect object, as indicated at block 1018.

With reference to FIG. 11, a flow diagram is illustrated which shows amethod 1100 for processing digital image data utilizing avertically-oriented Effect graph, the digital image data being processedfrom a bottom of the Effect graph to a top of the Effect graph, inaccordance with an embodiment of the present invention. Initially, asindicated at block 1110, a vertically-oriented Effect graph is received.The Effect graph includes at first and second Effect objects.Subsequently or simultaneously, digital image data to be processedutilizing the Effect graph is received, as indicated at block 1112.Subsequently, a first processing pass of at least a portion of theEffect graph including the first and second Effect objects is performed,as indicated at block 1114. Next, as indicated at block 1116, at least aportion of the digital image data is cached in an image bufferassociated with the output of the first Effect object, as indicated atblock 1116.

Subsequently, one or more user modifications in association with thesecond Effect object are detected, as indicated at block 1118. Next, asindicated at bock 1120, a second processing pass of the Effect graph isperformed, the second processing pass extending from the image bufferassociated with the output of the Effect object.

Turning now to FIG. 12, a flow diagram is illustrated which shows amethod 1200 for processing digital image data utilizing an Effect graph,in accordance with an embodiment of the present invention. Initially, asindicated at block 1210, an Effect graph is received. The Effect graphincludes a plurality of Effect objects each having an image bufferassociated therewith for storing digital image data. Subsequently orsimultaneously, as indicated at block 1212, digital image data to beprocessed utilizing the Effect graph is received. Next, a pre-renderingpass of the Effect graph is performed, as indicated at block 1214.Subsequently, the digital image data is processed utilizing the Effectgraph, as indicated at block 1216.

With reference to FIG. 13, a flow diagram is illustrated which shows amethod 1300 for processing digital image data utilizing an Effect graph,in accordance with an embodiment of the present invention. Initially, asindicated at block 1310, an Effect graph is received. The Effect graphincludes a plurality of Effect objects each having an image bufferassociated therewith for storing digital image data. Subsequently orsimultaneously, digital image data to be processed utilizing the Effectgraph is received, as indicated at block 1312. Next, it is determined ifsufficient memory is available in the GPU associated with the Effectgraph to process the digital image data. This is indicated at block1314. If there is sufficient memory available in a GPU associated withthe Effect graph to process the digital image data, the digital imagedata is processed utilizing the GPU, as indicated at block 1316.However, if it is determined that there is not sufficient memoryavailable in the GPU to process the digital image data, the digitalimage data is processed, at least in part, utilizing a CPU associatedwith the Effect graph. This is indicated at block 1318.

Turning now to FIG. 14, a flow diagram is illustrated which shows amethod 1400 for creating an Effect graph for processing of digital imagedata, in accordance with an embodiment of the present invention.Initially, as indicated at block 1410, one or more Effect Layers areprovided, each Effect Layer having a logical Effect sub-graph associatedtherewith that includes a plurality of logical Effects. Subsequently,input regarding the processing of digital image data is received, asindicated at block 1412. Such input may include, by way of example only,the type of main Effect included in the Layer, the settings applied tothe Layer interface (e.g., opacity equal to 100%), and other settingsand properties. Subsequently, as indicated at block 1414, the impact theinput received is to have on each of the logical Effects upon processingof the digital image data is determined. Next, a physical Effectsub-graph is created in accordance with the determined impact, asindicated at block 1416. An image may then be rendered utilizing thephysical Effect sub-graph, as indicated at block 1418.

With reference to FIG. 15, a flow diagram is illustrated which shows amethod for processing digital image data utilizing an Effect graph, inaccordance with an embodiment of the present invention. Initially, asindicated at block 1510, an Effect graph having at least one mask Effectobject is received, the mask Effect object being associated with anEffect Layer. Subsequently or simultaneously, digital image data isreceived, as indicated at block 1512. Next, as indicated at block 1514,input regarding selection of at least a portion of the digital imagedata received that satisfies a criteria is received. Subsequently, asindicated at block 1516, the at least one mask Effect object is appliedto the digital image data received.

In one embodiment, the method of claim 15 may include further processingsteps as shown in FIG. 16. Initially, as indicated at block 1610, inputis received regarding modification of the digital image data.Subsequently, the digital image data is modified in accordance with theinput received to create modified digital image data, as indicated atblock 1612. Next, as indicated at block 1614, the at least one maskEffect object is applied to the modified digital image data and the atleast one mask Effect object is automatically modified to satisfy thecriteria.

With reference to FIG. 17, a flow diagram is illustrated which shows amethod 1700 for processing digital image data utilizing avertically-oriented Effect graph, the digital image data being processedfrom a bottom of the Effect graph to a top thereof, in accordance withan embodiment of the present invention. Initially, as indicated at block1710, a vertically-oriented Effect graph is received, the Effect graphhaving a first transformation Effect object located at a first verticallocation therein. Subsequently, as indicated at block 1712, it isdetermined whether the first transformation Effect object may be appliedto the digital image data at a second vertical location nearer thebottom of the Effect graph than the first vertical location. If it isdetermined that the first transformation Effect object may not beapplied to the digital image data at the second vertical location, thefirst transformation Effect object is applied at the first verticallocation, as indicated at block 1714.

If, however, it is determined that the first transformation Effectobject may be applied to the digital image data at the second verticallocation, it is next determined whether there is a transformation Effectobject located at the second vertical location. This is indicated atblock 1716. If there is no transformation Effect object located at thesecond vertical location, the first transformation Effect object isapplied at the second vertical location rather than the first verticallocation, as indicated at block 1718. If, however, it is determined thatthere is a transformation Effect object located at the second verticallocation, the first and second transformation Effect objects arecumulatively applied at the second vertical location, as indicated atblock 1720.

Turning now to FIG. 18, a flow diagram is illustrated which shows amethod 1800 for processing digital image data utilizing avertically-oriented Effect graph, in accordance with an embodiment ofthe present invention. Initially, as indicated at block 1810, digitalimage data having an initial transformation value is received.Subsequently or simultaneously, an Effect graph having at least onetransformation Effect object is received, as indicated at block 1812.Next, the digital image data received is transformed at thetransformation Effect object from the initial transformation value to afirst transformed transformation value, as indicated at block 1814.Subsequently, the digital image data is transformed at thetransformation Effect object from the initial transformation value to asecond transformed transformation value, as indicated at block 1816.

In view of the above, an embodiment of the present invention is directedto a system for editing digital image data. The system includes aGraphic Processing Unit (GPU) having GPU memory associated therewith,and an editing component for non-destructively editing at least aportion of the digital image data utilizing the GPU memory. In oneembodiment, the editing component is capable of non-destructivelyediting at least a portion of the digital image data received utilizingone of the GPU memory and a combination of the GPU memory and the CPUmemory.

The system may further include an Effect graph receiving component forreceiving a plurality of Effect objects that are wired together to forman Effect graph, the Effect graph being capable of being utilized forediting the digital image data. Additionally, the system may furthercomprise a data receiving component for receiving the digital image datato be edited.

In another embodiment, the present invention is directed to a method forprocessing digital image data utilizing an Effect graph. The methodincludes receiving a plurality of Effect objects and wiring at least aportion of the plurality of Effect objects together into the Effectgraph, wherein the Effect graph is capable of being wired utilizing atleast one of CPU memory and GPU memory. In one embodiment, processingthe digital image data received may include non-destructively processingat least a portion of the digital image data. The method may furthercomprise receiving digital image data to be processed and/or processingthe digital image data received utilizing the Effect graph, as desired.Still further, the method may include storing the digital image data inassociation with an image buffer in at least one of the CPU memory andthe GPU memory.

In an additional embodiment, the present invention is directed to asystem for processing digital image data. The system includes a datareceiving component for receiving digital image data to be processed, anEffect graph obtaining component for obtaining an Effect graph to beutilized in processing the digital image data received, the Effect graphincluding a plurality of Effect objects, and a processing component forprocessing the digital image data utilizing the Effect graph and atleast one of CPU memory and GPU memory. In one embodiment, the Effectgraph obtaining component may be capable of wiring at least a portion ofthe plurality of Effect objects together to create the Effect graph. Inanother embodiment, the processing component is capable ofnon-destructively processing at least a portion of the digital imagedata utilizing the Effect graph and at least one of the GPU memory and acombination of the GPU memory and the CPU memory. If desired, the systemmay further include a storage component for storing the digital imagedata in at last one of the CPU memory and the GPU memory.

In an additional embodiment, the present invention relates to a methodfor processing digital image data utilizing a vertically-oriented Effectgraph, the digital image data being processed from a bottom of theEffect graph to a top of the Effect graph. The method includes receivingthe vertically-oriented Effect graph, wherein the Effect graph includesan Effect object located at a junction initiating two divergent paths,receiving digital image data to be processed utilizing the Effect graph,performing a first processing pass of the Effect graph from the bottomto the top thereof along a first of the two divergent paths, and cachingat least a portion of the digital image data in an image bufferassociated with the output of the Effect object during processing.

In another embodiment, the present invention is directed to a method forprocessing digital image data utilizing a vertically-oriented Effectgraph, the digital image data being processed from a bottom of theEffect graph to a top of the Effect graph. The method includes receivingthe vertically-oriented Effect graph, wherein the Effect graph includesat least a first Effect object and a second Effect object, the secondEffect object being nearer the top of the Effect graph than the firstEffect object, receiving digital image data to be processed utilizingthe Effect graph, performing a first processing pass of at least aportion of the Effect graph that includes the first and second Effectobjects, and caching at least a portion of the digital image data in animage buffer associated with the output of the first Effect object. Themethod may further comprise detecting user modification in associationwith the second Effect object.

In another embodiment, the present invention is directed to a system forprocessing digital image data utilizing a vertically-oriented Effectgraph, the digital image data being processed from a bottom of theEffect graph to a top of the Effect graph. The system includes a firstreceiving component for receiving the vertically-oriented Effect graph,wherein the Effect graph includes an Effect object located at a junctioninitiating two divergent paths. The system further includes a secondreceiving component for receiving digital image data to be processedutilizing the Effect graph, a first processing component for performinga processing pass of the Effect graph from the bottom to the topthereof, a caching component for caching at least a portion of thedigital image data in a buffer associated with the output of the Effectobject during processing, and a second processing component forperforming a second processing pass of the Effect graph in accordancewith one of the two divergent paths, wherein the second processing passextends from the image buffer associated with the output of the Effectobject.

In another embodiment, the present invention relates to a method forprocessing digital image data utilizing an Effect graph. The methodincludes receiving the Effect graph, wherein the Effect graph includes aplurality of Effect objects each having an image buffer associatedtherewith for storing digital image data, receiving digital image datato be processed utilizing the Effect graph, and performing apre-rendering pass of the Effect graph prior to processing the digitalimage data.

In yet another embodiment, the present invention relate to a method forprocessing digital image data utilizing an Effect graph. The methodincludes receiving the Effect graph, wherein the Effect graph receivedincludes a plurality of Effect objects each having an image bufferassociated therewith for storing digital image data, receiving digitalimage data to be processed utilizing the Effect graph, and determiningif sufficient memory is available in a Graphics Processing Unit (GPU)associated with the Effect graph to process the digital image data.

Still further, an embodiment of the present invention relates to amethod for processing digital image data utilizing an Effect graph. Themethod includes receiving the Effect graph, wherein the Effect graphincludes a plurality of Effect objects each having an image bufferassociated therewith for storing digital image data, receiving digitalimage data to be processed utilizing the Effect graph, and determiningif one or more of the image buffers associated with the plurality ofEffect objects is to be cached during processing of the digital imagedata.

In another embodiment, the present invention relates to a method forcreating an Effect graph for processing of digital image data. Themethod includes providing one or more Effect Layers, each Effect Layerhaving a logical Effect sub-graph associated therewith that includes aplurality logical Effects, receiving input regarding processing of thedigital image data, determining the impact the input received is to haveon each logical Effect upon processing of the digital image data, andcreating a physical Effect sub-graph in accordance with the determinedimpact.

In yet another embodiment, the present invention relates to a method forprocessing digital image data utilizing an Effect graph. The methodincludes receiving the Effect graph, the Effect graph having at leastone mask Effect object associated with an Effect Layer, receivingdigital image data, receiving input regarding selection of at least aportion of the digital image data that satisfies a criteria, andapplying the at least one mask Effect object to the digital image datareceived.

In still another embodiment, the present invention relates to a systemfor processing digital image data. The system includes a providingcomponent for providing one or more Effect Layers, each Effect Layerhaving a logical Effect sub-graph associated therewith that includesthree or more logical Effects, a transforming component for transformingthe logical Effect sub-graph into a physical Effect sub-graph inaccordance with an impact each of the logical Effects is to have uponprocessing of the digital image data, and a rendering component forrendering an image utilizing the physical Effect sub-graph. The systemmay further comprise a converting component for converting the extracteddigital image data into an analytical result and/or a forwardingcomponent for forwarding the analytical result to the main Effect, ifdesired.

In yet another embodiment, the present invention relates to a method forprocessing digital image data utilizing a vertically-oriented Effectgraph, the digital image data being processed from a bottom of theEffect graph to a top thereof. The method includes receiving thevertically-oriented Effect graph, the Effect graph having a firsttransformation Effect object located at a first vertical locationtherein and determining whether the first transformation Effect objectmay be applied to the digital image data at a second vertical location,the second vertical location being nearer the bottom of the Effect graphthan the first vertical location. If it is determined that the firsttransformation Effect object may be applied to the digital image data atthe second vertical location, the method further may further includeapplying the first transformation object at the second vertical locationrather than at the first vertical location. If, however, it isdetermined that the first transformation Effect object may not beapplied to the digital image data at the second vertical location, themethod may further include applying the first transformation Effectobject at the first vertical location.

In an additional embodiment, the present invention relates to a systemfor processing digital image data utilizing a vertically-oriented Effectgraph. The system includes a receiving component for receiving thevertically-oriented Effect graph, the Effect graph having a firsttransformation Effect object located at a first vertical locationtherein, a determining component for determining whether the firsttransformation Effect object may be applied to the digital image data ata second vertical location, the second vertical location being nearerthe bottom of the Effect graph than the first vertical location, and anapplying component for applying the first transformation Effect objectat the second vertical location rather than at the first verticallocation if it is determined that the first transformation Effect objectmay be applied at the second vertical location.

In a further embodiment, the present invention relates to a method forprocessing digital image data utilizing a vertically-oriented Effectgraph. The method includes receiving digital image data having aninitial transformation value, receiving the Effect graph, the Effectgraph having at least one transformation Effect object, transforming thedigital image data received at the at least one transformation Effectobject from the initial transformation value to a first transformedtransformation value, and transforming the digital image data receivedat the at least one transformation Effect object from the initialtransformation value to a second transformed transformation value.

Embodiments of the present invention further relate to one or morecomputer readable media having computer-executable instructions forperforming the methods herein described, as well as computers programmedto perform the methods.

The present invention has been described in relation to particularembodiments, which are intended in all respects to be illustrativerather than restrictive. Alternative embodiments will become apparent tothose of ordinary skill in the art to which the present inventionpertains without departing from its scope.

From the foregoing, it will be seen that this invention is one welladapted to attain all the ends and objects set forth above, togetherwith other advantages which are obvious and inherent to the system andmethod. It will be understood that certain features and subcombinationsare of utility and may be employed without reference to other featuresand subcombinations. This is contemplated by and is within the scope ofthe claims.

1. A system for editing digital image data, comprising: a GraphicsProcessing Unit (GPU) having GPU memory associated therewith; and anediting component for non-destructively editing at least a portion ofthe digital image data utilizing the GPU memory.
 2. The system of claim1, further comprising an Effect graph receiving component for receivinga plurality of Effect objects that are wired together to form an Effectgraph, wherein the Effect graph is capable of being utilized for editingthe digital image data.
 3. The system of claim 1, further comprising adata receiving component for receiving the digital image data to beedited.
 4. The system of claim 1, further comprising a CentralProcessing Unit (CPU) having CPU memory associated therewith, whereinthe GPU is integrated with the CPU.
 5. The system of claim 4, whereinthe editing component is capable of non-destructively editing at least aportion of the digital image data received utilizing one of the GPUmemory and a combination of the GPU memory and the CPU memory.
 6. Amethod for processing digital image data utilizing an Effect graph, themethod comprising: receiving a plurality of Effect objects; and creatingat least a portion of the plurality of Effect objects together into theEffect graph, wherein the Effect graph is capable of being createdutilizing at least one of Central Processing Unit (CPU) memory andGraphics Processing Unit (GPU) memory.
 7. The method of claim 6, furthercomprising receiving digital image data to be processed.
 8. The methodof claim 7, further comprising processing the digital image datareceived utilizing the Effect graph.
 9. The method of claim 8, whereinprocessing the digital image data received comprises non-destructivelyprocessing at least a portion of the digital image data received. 10.The method of claim 6, wherein each of the plurality of Effect objectsis associated with an image buffer, and wherein the method furthercomprises storing the digital image data in at least one of the CPUmemory and the GPU memory in association with at least one of the imagebuffers.
 11. The method of claim 10, wherein the CPU memory isassociated with a CPU and the GPU memory is associated with a GPU, andwherein each image buffer is capable of being passed between the GPU andthe CPU automatically and without user intervention.
 12. The method ofclaim 10, wherein each image buffer is capable of being convertedbetween a first pixel data format and a second pixel data formatautomatically and without user intervention.
 13. One or more computerreadable media having computer-executable instructions for performingthe method of claim
 6. 14. A computer programmed to perform the methodof claim
 6. 15. A system for processing digital image data, the systemcomprising: a data receiving component for receiving digital image datato be processed; an Effect graph obtaining component for obtaining anEffect graph to be utilized in processing the digital image datareceived, the Effect graph including a plurality of Effect objects; anda processing component for processing the digital image data utilizingthe Effect graph and at least one of Central Processing Unit (CPU)memory and Graphics Processing Unit (GPU) memory.
 16. The system ofclaim 15, wherein the Effect graph obtaining component is capable ofwiring at least a portion of the plurality of Effect objects together tocreate the Effect graph.
 17. The system of claim 15, wherein theprocessing component is capable of non-destructively processing at leasta portion of the digital image data utilizing the Effect graph and atleast one of the GPU memory and a combination of the GPU memory and theCPU memory.
 18. The system of claim 15, wherein each of the plurality ofEffect objects is associated with an image buffer, and wherein thesystem further comprises a storage component for storing the digitalimage data in at least one of the CPU memory and the GPU memory.
 19. Thesystem of claim 18, wherein the CPU memory is associated with a CPU andthe GPU memory is associated with a GPU, and wherein each image bufferis capable of being passed between the GPU and the CPU automatically andwithout user intervention.
 20. The system of claim 18, wherein eachimage buffer is capable of being converted between a first pixel dataformat and a second pixel data format automatically and without userintervention.